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Executive  Summary 

Major  technical  advancements  have  been  made  on  two  fronts  outlined  in  our  original  pro¬ 
posal: 

•  We  developed,  analyzed,  and  implemented  a  robust  software  framework,  the  model 
management  framework  (MMF),  for  using  surrogates  to  optimize  computationally  ex¬ 
pensive  nonlinear  objective  functions.  This  software  implementation  has  been  used  to 
investigate  the  effectiveness  of  the  MMF  approach  when  applied  first  to  several  aca¬ 
demic  test  problems  and  then  to  a  realistic  engineering  design  problem  from  Boeing. 
This  conforms  with  our  stated  goal  in  the  proposal  that  we  would  work  closely  with 
Boeing  to  make  sure  that  our  software  applies  to  real  problems.  The  implementa¬ 
tion  (in  Fortran  90)  is  publicly  available  and  has  been  transferred  to  our  collaborators 
at  Boeing  where  it  will  be  developed  to  investigate  such  problems  as  the  design  of 
helicopter  rotor  blades,  jet  engine  nozzles,  and  wings. 

•  We  developed  extensions  to  and  analysis  for  pattern  search  methods  for  nonlinear 
optimization.  We  have  been  able  to  develop  variants  to  handle  both  bound  and  general 
linear  constraints,  as  well  as  to  develop  a  rigorous  understanding  of  the  minimal  amount 
of  information  that  must  be  generated  at  a  single  iteration  to  ensure  progress  of  the 
optimization  process.  The  analysis  for  bound  constraints,  as  well  as  the  understanding 
of  the  minimal  amount  of  information  that  must  be  generated  a  single  iteration,  form 
the  basis  for  the  convergence  analysis  of  the  MMF.  The  extensions  for  general  linear 
constraints,  which  also  allows  for  a  more  frugal  implementation  for  bound  constraints, 
is  expected  to  be  incorporated  into  our  MMF  software  framework  in  the  near  future. 

Outcomes  on  both  these  fronts  exceeded  the  expectations  outlined  in  our  original  proposal 
and  resulted  in  multiple  papers,  several  reports,  and  a  dissertation. 

In  addition  to  the  two  main  objectives  outlined  above,  the  grant  was  also  used  to  support 
the  development  of  multiobjective  optimization  tools.  The  development  of  these  tools  forms 
the  basis  of  the  dissertation  of  Indraneel  Das.  Boeing  is  beginning  to  apply  these  tools  to 
some  of  their  design  problems.  So  far,  no  details  are  available,  but  we  have  been  promised 
some  success  stories  to  pass  on  to  AFOSR  and  DOE,  the  sources  of  graduate  student  support 
for  Dr.  Das,  who  is  now  with  IBM. 
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Virginia  Torczon: 

•  Colloquium  talk:  “Pattern  Search  Methods:  Theory  and  Practice”.  Department  of 
Computer  Science,  College  of  William  &  Mary;  March  17,  1995;  Williamsburg,  Vir¬ 
ginia. 

•  Invited  talk:  “A  Specific  Algorithm  for  Managing  Approximation  Models  in  Optimiza¬ 
tion”.  CRPC  Research  Symposium  and  Annual  Meeting,  Rice  University;  March  23, 
1995;  Houston,  Texas. 

•  Colloquium  talk:  “Pattern  Search  Methods  for  Nonlinear  Optimization”.  Department 
of  Computer  and  Mathematical  Sciences,  University  of  Houston — Downtown;  March 
29,  1995,  Houston,  Texas. 

•  Colloquium  talk:  “Pattern  Search  Methods:  Theory  and  Practice”.  Department  of 
Computational  and  Applied  Mathematics,  Rice  University;  April  17,  1995;  Houston, 
Texas. 

•  Invited  article:  “Pattern  search  methods  for  nonlinear  optimization”.  SIAG/OPT 
Views-and-News:  A  Forum  for  the  SIAM  Activity  Group  on  Optimization,  No.  6; 
Spring,  1995. 

•  Colloquium  talk:  “Pattern  Search  Methods  for  Nonlinear  Optimization  on  Parallel 
Machines”.  Department  of  Computer  Science,  Old  Dominion  University;  September 
27,  1995;  Norfolk,  Virginia. 

•  Colloquium  talk:  “Pattern  Search  Methods  for  Nonlinear  Optimization  on  Parallel  Ma¬ 
chines”.  Operations  Research  Interest  Group,  College  of  William  and  Mary;  November 
14,  1995;  Williamsburg,  Virginia. 

•  Colloquium  talk:  “Pattern  Search  Methods  for  Nonlinear  Optimization  on  Parallel 
Machines”.  Department  of  Computer  Science,  Virginia  Polytechnic  Institute  and  State 
University;  March  6,  1996;  Blacksburg,  Virginia. 

•  Invited  talk:  “Managing  Approximation  Models  in  Optimization”.  INFORMS  Confer¬ 
ence  on  Information  Systems  &  Technology;  May  7,  1996;  Washington,  D.C. 

•  Plenary  talk:  “Direct  Search  Methods”.  Fifth  SIAM  Conference  on  Optimization;  May 
21,  1996;  Victoria,  British  Columbia,  Canada. 

•  Plenary  talk:  “Managing  Approximation  Models  in  Optimization”.  ICASE  Workshop 
on  Approximation;  August  14,  1996;  NASA  Langley  Research  Center,  Hampton,  Vir¬ 
ginia. 
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•  Invited  talk:  “A  Pattern  Search  Approach  to  Managing  Approximation  Models  in 
Engineering  Optimization”.  INFORMS  Atlanta  Fall  1996;  November  4,  1996;  Atlanta. 

•  Contributed  talk:  “From  Evolutionary  Operation  to  Parallel  Direct  Search:  Pattern 
Search  Algorithms  for  Numerical  Optimization”.  Interface  ’97;  May  16,  1997;  Houston. 

•  Invited  short  Course:  “A  Parallel  Optimization  Primer”  (with  Robert  Michael  Lewis). 
CRPC  Annual  Research  Meeting;  May  20,  1997;  Houston. 

•  Invited  short  course:  “A  Parallel  Optimization  Primer”  (with  Robert  Michael  Lewis). 
Aeronautical  Systems  Center,  Wright-Patterson  Air  Force  Base;  May  29,  1997;  Dayton, 
Ohio. 

•  Contributed  talk:  “Pattern  Search  Methods  for  Engineering  Optimization” .  Electronic 
Prototyping,  Air  Force  Office  of  Scientific  Research;  May  29,  1997;  Blacksburg,  Vir¬ 
ginia. 

•  Invited  talk:  “Rank  Ordering  and  Positive  Bases  in  Pattern  Search  Methods”.  SIAM 
Annual  Meeting;  July  17,  1997;  Palo  Alto,  California. 

•  Contributed  talk:  “Robust  Derivative-Free  Methods  for  Linearly  Constrained  Mini¬ 
mization”.  International  Symposium  on  Mathematical  Programming;  August  26,  1997; 
Lausanne. 

M.  W.  Trosset: 

•  “Some  Fundamental  Ideas  in  Stochastic  Optimization”.  Departments  of  Statistics  and 
Computational  &  Applied  Mathematics,  Rice  University;  November  27,  1995;  Houston. 

•  “General  Issues  in  the  Optimization  of  Expensive  Functions” .  Boeing/IBM/Rice  Work¬ 
shop  on  Approximation  Models,  Rice  University;  February  22,  1996;  Houston. 

•  Plenary  Lecture:  “Numerical  Algorithms  for  Multidimensional  Scaling”.  20th  Annual 
Conference  of  Gesellschaft  fur  Klassifikation  e.V.;  March  7,  1996;  Freiburg,  Germany. 

•  “Computational  Methods  for  Stochastic  Minimum  Distance  Estimation” .  Department 
of  Operations  Research,  Naval  Postgraduate  School;  April  8,  1996;  Monterey,  Califor¬ 
nia. 

•  “Computational  Methods  for  Stochastic  Minimum  Distance  Estimation”.  Department 
of  Systems  &  Industrial  Engineering,  University  of  Arizona;  May  2,  1996;  Tucson, 
Arizona. 

•  “Local  Quadratic  Models  in  Stochastic  Optimization”  (Minisymposium  on  Beyond 
Taylor  Series  Approximations:  The  Use  of  Alternative  Models  in  Nonlinear  Program¬ 
ming”.  Fifth  SIAM  Conference  on  Optimization;  May  23,  1996;  Victoria,  British 
Columbia,  Canada. 
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•  Short  Course:  “Optimization  Problems  in  Multidimensional  Scaling.  Workshop  on 
Classification  and  Optimization  in  Statistics”.  Rice  Institute  for  Mathematical  Sci¬ 
ences,  Rice  University;  July  15-19,  1996;  Houston. 

•  “General  Issues  in  the  Optimization  of  Expensive  Functions”.  Workshop  on  Approxi¬ 
mation  Models,  Institute  for  Computer  Applications  in  Science  &  Engineering;  August 
15,  1996;  Langley,  Virginia. 

•  “Data  Box  Algorithms  for  Molecular  Conformation”.  Fall  Symposium  5:  Computa¬ 
tional  Biology — Methods  in  Biomolecular  Imaging,  American  Society  for  Biochemistry 
&  Molecular  Biology;  October  25-28,  1996;  Whistler,  British  Columbia,  Canada. 

•  “On  the  Fundamental  Role  of  Interior-Point  Methodology  in  Constrained  Optimiza¬ 
tion”.  Fall  Symposium  5:  Computational  Biology — Methods  in  Biomolecular  Imag¬ 
ing,  American  Society  for  Biochemistry  &  Molecular  Biology;  October  25-28,  1996; 
Whistler,  British  Columbia,  Canada. 

•  “New  Extensions  of  Classical  Multidimensional  Scaling”.  Department  of  Biostatistics, 
University  of  Colorado  Health  Science  Center;  November  14,  1996;  Denver. 

•  “New  Extensions  of  Classical  Multidimensional  Scaling”.  Department  of  Mathematics, 
University  of  Colorado  at  Denver;  November  18,  1996;  Denver. 

•  “New  Extensions  of  Classical  Multidimensional  Scaling”.  Department  of  Statistics, 
Virginia  Polytechnic  Institute  &  State  University;  December  5,  1996;  Blacksburg,  Vir¬ 
ginia. 

•  “Inferring  Molecular  Conformation  from  Interatomic  Distances”.  W.M.  Keck  Center 
for  Computational  Biology,  Rice  University;  February  21,  1997;  Houston. 

•  “Numerical  Optimization  Using  Computer  Experiments”.  Department  of  Statistics, 
Rice  University;  April  21,  1997;  Houston. 

•  “Applications  of  Multidimensional  Scaling  to  Molecular  Conformation”.  Interface  ’97; 
May  15-17,  1997;  Houston. 

•  “Inferring  Molecular  Structure  from  Interatomic  Distances”.  Statistics  in  Molecular 
Biology,  Joint  Summer  Research  Conference  in  the  Mathematical  Sciences;  June  22-26, 
1997;  Seattle. 

•  Short  Course:  “Introduction  to  Dimension  Reduction” .  Workshop  on  Statistical  Func¬ 
tion  Estimation,  Image  Understanding  and  Optimization;  Rice  Institute  for  Mathe¬ 
matical  Sciences,  Rice  University;  June  2-13,  1997;  Houston. 

•  “Approximation  and  Modelling  in  Stochastic  Optimization”.  Workshop  on  Approxi¬ 
mations  and  Surrogates,  Institute  for  Computer  Applications  in  Science  &  Engineering; 
July  21-23,  1997;  Langley,  Virginia. 
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•  “On  the  Existence  of  Nonglobal  Minimizers  of  the  STRESS  Criterion  for  Metric  Mul¬ 
tidimensional  Scaling”.  Joint  Statistical  Meetings;  August  10-14,  1997;  Anaheim, 
California. 


Consultative  and  Advisory  Functions: 

J.  E.  Dennis 

•  Organized  a  visit  to  Rice  University  on  March  20,  1996  at  the  request  of  four  young 
scientists  from  Lever  Brothers  for  the  purpose  of  their  becoming  acquainted  with  new 
technologies. 

•  Arranged  for  Jennifer  Rich,  one  of  his  Engineering  Optimization  students,  to  do  a  class 
project  at  Texaco  on  optimal  blending  of  unleaded  gasoline;  April,  1996. 

•  Met  with  Dr.  Francisco  Brana  of  Shell  Development  several  times  to  discuss  MDO 
approaches  to  chemical  process  control;  1996. 

•  Chaired  of  the  External  Review  in  Computer  Science,  Duke  University;  Spring,  1996. 

•  Interdisciplinary  Mathematics  Ph.D.  Program  Advisory  Committee,  University  of  Puerto 
Rico;  October,  1996. 

•  Applied  Mathematics  Center  Advisory  Committee,  University  of  Florida;  November, 

1996. 

•  NSF  Renewal  Site  Visit  Team  to  IMA,  University  of  Minnesota;  February  1997. 

•  University  of  Chicago  Computer  Science  Review  Committee;  December  15-17,  1997. 

•  One  of  4  Academics  on  DOE  Panel  to  define  DOE2000  Program,  Lawrence  Berkeley 
Labs;  Feb  9-10,  1998. 

•  Organized  Boeing/IBM/Rice  collaboration  workshop  in  Houston,  (with  Teresa  Parks); 
Feb  19-20,  1998. 

Virginia  Torczon 

•  Consultant  in  Residence,  ICASE,  NASA  Langley  Research  Center,  Hampton,  Virginia, 
October,  1995-present. 

•  Invited  Participant:  1997  Petaflops  Algorithms  Workshop  (PAL  ’97)  (jointly  sponsored 
by  DARPA,  DOE,  NASA,  NSA,  and  NSF);  April  13-18,  1997;  Williamsburg,  Virginia. 

•  Served  on  two  National  Science  Foundation  Scientific  Computing  Research  Environ¬ 
ments  for  the  Mathematical  Sciences  (SCREMS)  panels,  April,  1996  and  February 

1997. 
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Douglas  Moore 

•  Consulted  with  Juan  Meza  of  Sandia  National  Laboratories  (CA)  on  issues  of  numerical 
software  design. 


Transitions: 

Earthquake  Simulation 

In  1996,  Rice  Professors  Steve  Cox  and  Joel  Conte  used  PDS  to  solve  for  the  first  time  an 
optimization  formulation  of  the  problem  of  retrofitting  existing  buildings  to  suppress  earth¬ 
quake  vibrations.  Conte  also  used  PDS  to  solve  an  inverse  problem  to  determine  earthquake 
source  location. 

Sandia  Labs  OPT++  Engineering  Optimization  System 

In  1996,  the  constraints  version  of  PDS(MDS)  was  incorporated  into  the  Sandia  Labs 
OPT++  Engineering  Optimization  System  of  Dr.  Juan  Meza  (510)294-2425  with  the  help 
of  David  Serafini  and  Doug  Moore.  Moore  collaborated  with  Meza  in  redesigning  the  object 
oriented  OPT++  system. 

Sandia  Chemical  Vapor  Deposition  Problem 

The  TRICE  modules  for  optimization  with  simulation  constraints  were  used  to  solve  a  chem¬ 
ical  vapor  deposition  problem  in  silicon  wafer  production.  This  code  performed  better  than 
another  interior  point  NLP  code  based  on  different  principles. 

Boeing  Parts  Nesting  System 

PDS  continues  to  be  used  in  the  Boeing  Parts  Nesting  System  for  Just-in-Time  manufactur¬ 
ing  of  aircraft  parts. 

PDS  Software 

The  latest  version  of  the  PDS  (Parallel  Direct  Search)  software  for  unconstrained  optimiza¬ 
tion  problems  will  be  installed  in  the  Network-Enabled  Optimization  System  (NEOS),  man¬ 
aged  by  the  Mathematics  and  Computer  Science  Division,  Argonne  National  Laboratory, 

Illinois,  at  the  invitation  of  Jorge  More. 

Torczon  has  been  invited  to  submit  PDS  for  inclusion  in  the  National  HPCC  Software 
Exchange  (NHSE),  a  program  to  actively  promote  software  sharing  and  reuse  within  and 
across  the  participating  agencies  (NIST,  AHCPR,  DARPA,  NOAA,  NASA,  EPA,  VA,  NIH, 

NSF,  ED,  NSA,  and  DOE)  in  the  federal  High  Performance  Computing  and  Communications 
(HPCC)  Program,  (http://www.nhse.org) 

Torczon  has  also  been  invited  to  place  PDS  in  PINEAPL,  a  European  project  to  develop 
parallel  numerical  library  software  for  industrial  applications,  managed  by  the  Numerical  Al¬ 
gorithms  Group  (NAG)  Ltd.,  United  Kingdom.  (http://extweb.nag.co.uk/projects/PINEAPL.html). 
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In  addition  to  including  the  parallel  variant  of  PDS  in  PINEAPL,  NAG  has  expressed  an 
interest  in  incorporating  the  sequential  variant  into  their  commercial  numerical  software 
libraries,  (http://www.nag.co.uk/numeric.html) 

Nozzle  Design 

Our  group  is  working  with  a  group  in  Boeing  Commercial  Airplane  Group  (BCAG)  to  reduce 
the  cycle  time  for  designing  nozzles.  A  nozzle  is  the  inside  part  of  the  engine  housing,  and  its 
design  is  affected  by  the  design  of  the  more  “upstream”  components.  This  means  that  many 
of  the  design  changes  involving  other  airplane  components  force  a  redesign  of  the  nozzle  as 
well. 

A  nozzle  design  is  specified  by  100  parameters,  90  of  which  are  fixed  by  other  considera¬ 
tions.  The  system  is  governed  by  a  2D  Navier  Stokes  coupled  with  NASTRAN,  a  commercial 
structures  code.  It  takes  about  3  hours  on  an  SGI  Challenge  to  get  one  function  value.  The 
current  length  of  a  design  cycle  is  two  weeks.  We  expect  to  reduce  that  to  approximately 
one  day. 

Contact:  Greg  Shubin  (425)  865-3516. 

Planform  Design 

Our  group  is  working  with  another  BCAG  group  that  does  planform  design.  Here,  they  are 
hoping  for  better  planforms,  not  just  reduced  cycle  time.  The  planform  is  the  shape  of  the 
wing  as  viewed  from  above.  It  is  a  tricky  design  problem  that  involves  a  couple  of  dozen 
variables  and  some  interesting  constraints.  For  example,  the  fuel  is  outboard  on  the  wings, 
and  the  wings  must  be  swept  back  for  performance.  One  constraint  is  that  the  plane  must 
not  fall  over  backwards  when  the  tanks  are  filled  while  the  plane  is  on  the  ground.  This 
problem  involves  multiple  objectives. 

Contact:  Greg  Shubin  (425)  865-3516. 

High-Speed  Cutting  Tool  Design 

High-speed  machine  tools  encounter  the  metal  at  over  300  mph.  At  that  speed,  the  metals 
are  not  fully  solid.  This  allows  more  complicated  parts  to  be  machined  in  a  single  piece. 
Many  large  airplane  parts  are  currently  assembled  from  components  that  were  machined 
separately.  If,  during  airplane  assembly,  one  of  these  parts  doesn’t  quite  fit,  work  stops  and 
a  supervisor  is  called  in  to  approve  the  use  of  shims  to  make  it  fit.  The  Boeing  plant  in 
Wichita  found  a  significant  reduction  in  the  need  for  this  “shimming”  when  large  parts  were 
machined  as  a  single  piece  using  a  high-speed  machine  tool. 

Since  this  is  a  new  technique,  no  one  yet  knows  how  to  design  a  really  good  cutting  tool. 
This  is  a  three-variable,  very  expensive  multiobjective  problem. 

Contact:  Greg  Shubin  (425)  865-3516. 

Helicopter  Rotor  Blade  Design 

This  is  the  problem  we  have  been  working  on  for  the  longest  time.  Wake  simulation  is  the 
hard  part  of  the  problem,  and  the  effects  at  the  tip  of  the  rotor  blade  are  the  most  difficult  to 
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simulate.  It  takes  about  4  hours  on  16  fat  nodes  of  an  SP2.  Our  group  at  Boeing  managed 
to  build  a  response  surface  model  for  the  spline  coefficients  of  the  output  of  the  full  potential 
rotor  discipline  as  a  function  of  the  spline  coefficients  of  the  input  to  that  code,  which  is 
the  output  from  the  coupled  simulation  involving  thermal  analysis,  structural  analysis,  and 
aerodynamics  analysis.  This  new  model  reduces  the  cost  to  minutes  and  the  bandwidth  of 
the  coupling  from  100s  to  10s.  We  will  be  experimenting  with  the  14  objective  functions 
they  want  to  trade  off  using  this  cheaper  code. 

Contact:  Greg  Shubin  (425)  865-3516. 

Model  Management  Framework  Software 

The  framework  software  has  been  transferred  to  our  collaborators  at  Boeing.  They  will 
use  the  framework  to  develop  model  management  algorithms  that  are  specialized  for  their 
applications  and  capabilities. 

Contact:  Greg  Shubin  (425)  865-3516. 

SLP  Software 

COPILOT  has  been  used  to  solve  a  problem  in  multidisciplinary  optimization  from  NASA 
Langley,  as  well  as  on  standard  benchmark  sets.  The  software  will  appear  shortly  on  the 
NEOS  server,  and  will  be  made  available  over  the  Internet. 

Contact:  Natalia  Alexandrov  (757)  864-7059 


Inventions,  or  Patent  Disclosures: 

None. 


Honors  /  Awards: 

J.  E.  Dennis 

•  President  of  the  Mathematical  Programming  Society. 

•  Member,  International  Program  Committee  of  the  Mathematical  Programming  Society 
(1987—). 

•  Founding  Editor-In-Chief  of  SIAM  Journal  on  Optimization. 

•  Served  on  and  chaired  numerous  panels  and  visiting  committees. 

•  Advisory  Editor,  Mathematics  of  Operations  Research ,  (1992 — ). 

•  Served  two  terms  on  the  SIAM  Council. 

•  Fulbright  Lecturer  to  Argentina. 
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Virginia  Torczon 

•  Invited  to  join  the  editorial  board  of  the  new  SIAM  Book  Series  in  Advances  in  Design 
and  Control,  John  A.  Burns,  Editor-in-Chief. 

•  Elected  Program  Director,  SIAM  Activity  Group  on  Optimization,  January  1,  1 998— 
December  31,  2000. 

•  Invited  to  organize  the  Cluster  on  Nonlinear  Programming  for  INFORMS  Seattle, 
October  25-28,  1998. 

•  Invited  to  serve  as  Member,  Organizing  Committee,  Sixth  SIAM  Conference  on  Opti¬ 
mization;  Atlanta,  Georgia;  May  10-12,  1999. 

•  Elected  to  SIAM  Council,  Society  for  Industrial  and  Applied  Mathematics;  January  1, 
1997-December  31,  1999. 

•  Appointed  to  SIAM  Committee  on  Committees  and  Appointments;  January  1,  1997- 
December  31,  1998. 

•  Co-chair,  Eighth  SIAM  Conference  on  Parallel  Processing  for  Scientific  Computing; 
Minneapolis,  Minnesota;  March  14-17,  1997. 

•  Appointed  to  the  Technical  Steering  Committee,  Optimization  and  Automatic  Differ¬ 
entiation,  Center  for  Research  in  Parallel  Computation;  effective  November  1996. 

•  Gave  plenary  address  Direct  Search  Methods  at  5th  SIAM  Conference  on  Optimization, 
May  1996. 

•  Member,  Tutorial  Committee,  Supercomputing  ’95,  November  1995. 

•  Secretary,  SIAM  Activity  Group  on  Supercomputing,  January  1,  1994-December  31, 
1996. 
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Final  Report  for  Grant  Number:  F49620-95-1-0210 
Managing  the  Choice  of  Surrogate  Variables  and  the  Use  of 
Approximation  Models  to  Optimize  Expensive  Functions 

The  Context  of  the  Research 

Mathematically,  the  problem  we  solve  can  be  expressed  as: 

minimize  f(x)  (1) 

subject  to  x  G  B  =  {x  |  a  <  x  <  6}, 

where  /  :  U"  ^  K  U  {oo},  a,  6  €  5Rn,  and  a  <  b  means  that  each  coordinate  satisfies  a;  <  6,-. 

Problem  (1)  is  an  optimization  problem  with  simple  bound  constraints.  Most  problems 
also  include  other  types  of  constraints.  Our  helicopter  examples  include  an  additional  linear 
inequality  constraint.  Usually,  such  constraints  are  managed  either  by  using  them  to  elimi¬ 
nate  one  of  the  variables  or  simply  by  assigning  a  large  function  value  to  infeasible  points, 
although  the  latter  approach  can  fail  in  theory.  Recent  analysis  for  pattern  search  methods 
[32]  eliminates  this  shortcoming  and  can  be  extended  to  our  model  management  framework. 
Managing  nonlinear  constraints,  especially  equality  constraints,  is  an  issue  that  we  have  not 
yet  addressed  in  the  model  management  framework. 

Some  key  properties  of  our  target  problems  are: 

1.  The  number  of  decision  variables  x  is  reasonably  small,  say  n  <  100. 

2.  It  is  impractical  to  accurately  approximate  derivatives  of  /. 

3.  The  routines  that  evaluate  f(x)  may  fail  for  some  feasible  x  at  the  same  cost  as  if  a 
value  had  been  obtained. 

4.  If  x  violates  any  of  the  bound  constraints,  then  f(x)  may  not  be  available. 

5.  The  computation  of  f(x)  is  very  expensive  and  the  values  obtained  may  have  few 
correct  digits. 

Although  the  number  of  optimization  variables  is  reasonably  small,  the  total  number  of 
variables  in  the  problem  usually  is  large.  Typically,  f(x)  is  expensive  to  evaluate  because 
there  are  large  numbers  of  ancillary  or  system  variables  that  must  be  determined  for  each 
choice  of  x  before  f(x)  can  be  evaluated.  For  the  helicopter  rotor  blade  problem,  x  specifies 
a  coupled  set  of  partial  differential  equations  (PDEs)  that  must  be  solved  in  order  to  obtain 
dependent  system  variables  that  are  then  used  to  evaluate  f(x).  The  coupling  of  PDEs 
via  some  iterative  method,  most  often  the  notoriously  unreliable  successive  substitution  ap¬ 
proach,  explains  the  third  of  our  assumptions,  since  the  method  may  run  for  many  iterations 
and  not  converge. 

The  second  and  third  of  our  intrinsic  assumptions  make  quasi-Newton  methods  difficult 
to  apply.  Choosing  a  finite-difference  step  size  to  approximate  derivatives  is  a  difficult  and 
crucial  part  of  getting  a  finite-difference  quasi-Newton  method  to  work.  This  difficulty  is 
compounded  by  the  fact  that  we  may  not  be  able  to  compute  the  function  value  at  the  step 


16 


size  selected.  We  hope  that  as  automatic  differentiation  technology  advances,  actual  deriva¬ 
tives  can  be  used  rather  than  finite-difference  approximations.  However,  the  last  assumption 
is  again  relevant  because  quasi-Newton  methods  are  affected  by  function  inaccuracies  [23]. 

If  we  discount  the  expense  of  evaluating  f(x ),  then  direct  search  methods  [51,  54,  53] 
avoid  many  of  the  difficulties  that  we  have  identified.  Indeed,  parallel  direct  search  (PDS) 
[21,  50]  solved  10,  31,  and  56  variable  instances  of  the  helicopter  rotor  blade  design  problem. 
Direct  search  methods  are  sampling  methods  and  so  typically  many  function  values  are 
required  to  solve  the  problem.  Furthermore,  the  “curse  of  dimensionality”  also  plays  a  major 
role;  problems  in  higher  dimensions  require  significantly  more  function  evaluations  than  do 
problems  in  lower  dimensions.  The  curse  of  dimensionality  can  be  ameliorated  somewhat 
by  exploiting  the  fact  that  PDS  is  “embarrassingly”  parallel  and  can  be  installed  easily  on 
any  parallel  or  distributed  computing  platform  that  supports  the  message-passing  interface 
(MPI). 

The  unifying  theme  of  this  project  has  been  that  we  wished  to  apply  a  generalized  pattern 
search  method  indirectly  using  surrogates  as  guides  to  make  adaptive  choices  as  to  where  to 
evaluate  the  objective  function.  Our  motivation  was  that  surrogates,  which  are  chosen  to  be 
inexpensive  to  evaluate,  could  be  used  to  avoid  as  much  as  possible  unnecessary  computation 
of  the  expensive  objectives.  A  major  goal  of  our  research  has  been  to  combat  the  curse  of 
dimensionality,  which  often  hinders  the  effectiveness  of  pattern  search  methods  when  applied 
to  problems  in  higher  dimensions.  However,  we  maintain  the  structure  of  pattern  search 
methods  to  ensure  our  method  is  robust. 

In  fact,  our  goal  was  realized.  Boeing  is  currently  most  interested  in  the  31-variable 
helicopter  problem — PDS  and  MMF  greatly  reduced  the  baseline  objective  function  value 
provided  by  our  collaborators  at  Boeing.  While  PDS  successfully  solved  this  problem,  the 
MMF  solved  it  with  fewer  evaluations  of  the  objective  (5465  and  237,  respectively).  In  ad¬ 
dition,  we  have  made  useful  extensions  to  the  pattern  search  methods  and  their  convergence 
theory — extensions  which  support  both  the  implementation  and  analysis  of  the  MMF. 


Summary  of  Results 

In  the  proposal  “Managing  the  Choice  of  Surrogate  Variables  and  the  Use  of  Approximation 
Models  to  Optimize  Expensive  Functions,”  we  proposed  research  in  three  basic  areas: 

1.  The  use  of  approximation  models  as  surrogates  for  expensive  functions  in  optimization; 

2.  Extensions  of  pattern  search  methods  to  handle  optimization  problems  with  bound 
constraints;  and 

3.  The  use  of  subspace/surrogate  variable  techniques  in  optimization. 

We  made  dramatic  progress  on  the  first  two  topics,  including  some  findings  that  had  not 
been  anticipated  by  our  original  proposal. 

The  third  topic  is  still  important,  but,  frankly,  both  our  model  management  framework 
software  (MMF)  and  our  parallel  direct  search  software  (PDS)  performed  much  better  and 
for  much  larger  problems  than  we  had  ever  anticipated.  Thus  progress  in  the  first  two  areas 
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took  precedence.  However,  the  time  will  come  when  new  approaches  will  be  needed  to  tackle 
even  larger  problems,  and  we  still  believe  that  our  subspace  approach  is  promising. 

In  this  report  we  summarize  accomplishments  pertaining  to  these  topics.  More  details 
can  be  found  in  the  references  [1,  7,  8,  9,  10,  22,  30,  31,  32,  41,  52,  55]. 

Optimization  using  Surrogate  Objectives 

This  was  the  major  thrust  of  F49620-95-1-0210.  Our  proposals  were  based  on  the  model 
management  framework  proposed  by  Dennis  and  Torczon  [22],  in  which  techniques  that  do 
not  rely  on  explicit  approximations  to  gradients  of  the  objective  function  were  employed. 
We  were  able  to  develop  a  simpler  and  more  direct  version  of  model  management,  which  is 
implemented  in  the  MMF  software. 

The  results  of  our  research  on  this  topic  are  summarized  in  the  dissertation  of  David 
Serafim  and  in  our  papers  [9]  and  [10];  they  include  a  through  discussion  and  analysis  of  our 
simpler  model  management  strategy.  The  remainder  of  Serafini’s  dissertation  is  devoted  to 
developing  a  software  prototype  that  is  sufficiently  flexible  to  accommodate  both  a  variety 
of  algorithms  for  the  optimization  and,  importantly,  a  variety  of  classes  of  objective  function 
approximations.  The  resulting  paper  [10]  is  undergoing  final  revision  now. 

The  MMF  software  has  produced  consistently  excellent  results  with  approximations  cho¬ 
sen  to  be  either  variable-order  multivariate  polynomial  interpolants  to  the  objective  function 
[20]  or  interpolants  from  the  design  and  analysis  of  computer  experiments  (DACE)  literature, 
which  are  in  wide  use  at  Boeing  [5,  6].  The  DACE  approximations  are  so-called  kriging  mod¬ 
els  which  originated  in  the  geophysics  community  but  have  gained  wide  currency  in  the  statis¬ 
tical  community  (see  [4,  5,  11,  12,  13,  16,  17,  26,  27,  29,  36,  38,  39,  40,  42,  45,  47,  56,  57,  58]). 
Trosset  and  Torczon  [55]  have  preliminary  numerical  results  to  suggest  that  even  a  greatly 
simplified  kriging  approximation  can  be  more  efficient  for  optimization  with  scarce  resources 
than  the  one-shot  experiment  approach  often  suggested  in  the  computer  experiment  litera¬ 
ture. 

Alexandrov,  Dennis,  Lewis  and  Torczon  [1]  proposed  an  analogous  management  strategy 
for  first-order  methods  (methods  that  do  explicitly  approximate  gradients)  and  extended  the 
convergence  theory  for  trust-region  methods  to  establish  global  convergence  to  a  local  solu¬ 
tion.  The  importance  of  this  work  is  that  it  shows  how  much  simpler  a  model  management 
strategy  can  be,  even  for  the  constrained  case,  if  the  surrogates  are  assumed  to  match  the 
derivatives  of  the  true  objective  and  constraint  functions. 

Pattern  search  methods 

Our  analysis  of  the  MMF  has  been  based  on  the  definition  and  convergence  analysis  of 
pattern  search  methods,  for  which  a  brief,  general,  less  technical  introduction  can  be  found 
in  [53]. 

Torczon  [52]  identified  a  common  structure  underlying  what  had  been  regarded  as  a  dis¬ 
parate  collection  of  search  techniques  based  on  practical  heuristics.  This  structure  defines  the 
class  of  pattern  search  methods,  for  which  Torczon  [52]  also  established  a  global  convergence 
theory  in  the  case  of  unconstrained  optimization.  Besides  providing  a  rigorous  foundation 
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for  the  folklore  that  pattern  search  methods  are  robust  in  practice,  the  convergence  anal¬ 
ysis  is  theoretically  interesting  because  of  its  close  relationship  to  the  extant  analysis  for 
quasi-Newton  methods  (see  [35]). 

Recent  research  papers  by  Lewis  and  Torczon  [30,  31,  32]  build  on  Torczon  [52]  .  Lewis 
and  Torczon  extended  the  convergence  analysis  to  the  case  of  bound-constrained  [30]  opti¬ 
mization  problems.  An  interesting  algorithmic  consequence  of  this  work  is  that  it  possible, 
with  only  simple  modifications  to  the  original  methods,  to  construct  pattern  search  methods 
that  are  guaranteed  to  converge  to  a  Karush-Kuhn-Tucker  point. 

Lewis  and  Torczon  [32]  further  extends  the  theory  to  the  case  of  general  linear  constraints. 
The  algorithmic  consequences  of  this  analysis  are  more  provocative  as  no  existing  pattern 
search  method  satisfies  the  conditions  necessary  to  guarantee  convergence  to  a  Karush-Kuhn- 
Tucker  point,  nor  can  one  be  obtained  from  simple  modifications  of  the  original  methods. 
They  were  encouraged  by  the  discovery  that  a  similar  line  of  analysis  by  May  [33]  led  to 
promising  numerical  results  when  implemented  and  tested. 

Lewis  and  Torczon  [31]  also  generalized  the  convergence  analysis  found  in  Torczon  [52]; 
the  effect  of  this  generalization  was  to  (almost)  halve  the  worst  case  cost  of  a  single  iteration 
of  a  pattern  search  method  in  the  unconstrained  case  by  reducing  the  maximum  number  of 
function  evaluations  required  from  2n  to  n  +  1.  In  the  case  of  bound  constraints,  the  worst 
case  cost  of  a  single  iteration  ranges  from  n  +  1  to  2n  function  evaluations,  depending  on  the 
number  of  constraints,  as  is  discussed  in  more  detail  in  [32], 

Subspace  techniques  in  optimization 

In  the  proposal  “Managing  the  Choice  of  Surrogate  Variables  and  the  Use  of  Approximation 
Models  to  Optimize  Expensive  Functions,”  we  suggested  applying  an  optimization  method 
to  solve  subproblems  posed  on  subspaces  of  lower  dimension  and  then  applying  the  method 
(or  possibly  even  a  different  optimization  method)  to  solve  the  full-dimensional  problem  on 
the  affine  hull  defined  by  the  current  iterate  and  the  solutions  found  for  the  subproblems. 

Subspace  techniques  would  be  useful  in  several  ways.  The  biggest  effect  of  dimension  on 
the  model  management  framework  is  that  the  construction  of  the  approximations  is  based 
on  sampling,  using  design  of  experiments  to  choose  the  interpolation  sites  that  define  the 
approximation.  The  number  of  points  required  quickly  escalates  with  dimension — and  the 
expensive  objective  has  to  be  evaluated  at  each  of  these  points.  When  we  use  interpola¬ 
tor  models  as  surrogates,  we  defer  the  curse  of  dimensionality  that  afflicts  pattern  search 
methods,  but  we  do  not  eliminate  it  altogether,  as  we  still  must  sample  the  search  space  to 
construct  an  initial  approximation.  Though  it  is  less  important,  the  problem  of  determining 
the  data  sites  requires  us  to  find  as  nearly  as  possible  the  global  solution  of  an  optimiza¬ 
tion  problem.  A  different  “global”  optimization  problem  must  be  solved  to  determine  the 
nonlinear  interpolant  which  we  will  use  as  an  approximation.  Global  optimization  is  nearly 
impossible  even  in  low  dimensions  and  global  techniques  becomes  completely  impractical  in 
higher  dimension.  Furthermore,  the  model  management  framework,  while  less  subject  to 
dimensional  effects  than  a  standard  pattern  search  technique  applied  directly  to  the  expen¬ 
sive  function,  has  some  dimensional  effect  near  a  solution.  However,  it  is  doubtful  that  the 
subspace  techniques  will  be  of  much  help  in  that  respect. 
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Fortunately,  our  original  concerns  about  the  efficacy  of  pattern  search  methods  when 
used  directly  to  solve  problems  with  large  numbers  of  variables  did  not  prevent  us  from 
effectively  tackling  design  problems  with  as  many  as  56  design  variables — far  larger  than 
we  had  originally  thought  possible.  In  fact,  pattern  search  methods  have  been  successfully 
applied  to  optimization  problems  with  as  many  as  256  variables.  We  still  believe  that  even 
larger  problems  could  be  solved  effectively  using  the  subspace  techniques  and  thus  expect 
this  subject  to  be  a  topic  of  future  research. 

Software 

There  are  two  distinct  software  products  from  this  research:  PDS  and  MMF.  Of  the  two 
products,  PDS  is  the  more  mature.  It  is  also  more  widely  applicable,  easier  to  apply,  and 
can  be  executed  on  either  a  sequential,  parallel,  or  distributed  computing  platform.  PDS  is 
widely  distributed  and  has  been  recognized  as  filling  an  important  niche  in  the  catalog  of 
optimization  software  tools.  On  the  other  hand,  MMF  is  a  very  new  software  product  that 
currently  exists  as  a  Fortran  90  prototype.  It  has  been  transferred  to  Lawrence  Berkeley 
Labs,  to  Sandia  Labs,  and  to  Boeing.  We  plan  to  transfer  it  to  Mobil  this  year.  The  lack 
of  high  quality  Fortran  90  compilers  for  various  computing  platforms  has  led  Doug  Moore 
to  begin  a  C++  reimplementation,  which  will  be  completed  by  fall  1998.  The  C++  version 
will  also  be  publicly  available. 

PDS 

Torczon  is  in  the  process  of  completing  a  major  revision  of  PDS,  a  software  package  for 
a  family  of  parallel  direct  search  methods  for  nonlinear  optimization  that  can  be  executed 
on  sequential,  distributed,  and  parallel  computing  platforms.  This  family  of  optimization 
algorithms  is  described  in  Dennis  and  Torczon  [21];  the  ideas  for  these  methods  builds  on 
the  work  found  in  Torczon  [48,  49]. 

The  original  release  of  PDS  was  written  using  machine-specific  libraries  or  compiler  di¬ 
rectives  for  implementations  on  a  limited  set  of  parallel  computing  platforms.  With  the  help 
of  David  Serafini,  the  new  release  of  the  software  now  makes  use  of  the  Message  Passing 
Interface  (MPI)  [44],  which  allows  for  generally  seamless  ports  between  all  parallel  and  dis¬ 
tributed  computing  platforms  that  support  MPI.  Once  the  external  documentation  has  been 
completed  and  the  implementation  has  passed  through  a  final  review,  it  will  be  resubmitted 
to  ACM  Transactions  on  Mathematical  Software  for  publication. 

PDS  is  also  garnering  increasing  interest  in  the  numerical  software  community.  Ser¬ 
afini  and  Torczon  were  invited  to  install  PDS  on  the  Network- Enabled  Optimization  System 
(NEOS)  Server  at  Argonne  National  Laboratory  [18]. 1  The  invitation  to  install  PDS  on 
NEOS  was  extended  both  because  PDS  provides  a  family  of  robust  techniques  for  handling 
optimization  problems  when  derivatives  are  unavailable  and  finite-difference  approximations 
are  unreliable  and  because  PDS  is  one  of  the  few  general-purpose,  “scalable,”  parallel  non¬ 
linear  optimization  software  packages. 

1URL :  http : / www . mcs . anl . gov/ot c/Server/ 
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For  the  same  reasons,  Torczon  has  also  been  invited  to  submit  PDS  for  inclusion  in  the 
Parallel  Industrial  Numerical  Applications  and  Portable  Libraries  (PINEAPL),  a  project 
of  the  European  Commission  to  develop  parallel  numerical  library  software  for  industrial 
applications.2  This  project  is  managed  by  the  well-regarded  Numerical  Algorithms  Group 
(NAG),  Ltd.,  United  Kingdom,  an  established  software  vendor  with  a  large  European  in¬ 
dustrial  customer  base  (as  well  as  a  U.S.  subsidiary  and  a  U.S.  customer  base).  In  addition 
to  including  the  parallel  variant  of  PDS  in  PINEAPL,  NAG  has  expressed  an  interest  in 
incorporating  the  sequential  variant  into  their  commercial  numerical  libraries.3  Final  deci¬ 
sions  will  be  made  once  the  revision  to  the  software  has  been  completed  and  NAG  has  had  a 
chance  to  exercise  the  extensive  validation  process  that  all  software  must  pass  to  be  included 
in  the  NAG  numerical  libraries. 

Finally,  Torczon  has  been  asked  to  submit  PDS  for  inclusion  in  the  National  HPCC  Soft¬ 
ware  Exchange  (NHSE),4  a  program  to  actively  promote  software  sharing  and  reuse  within 
and  across  the  twelve  participating  agencies5  in  the  federal  High  Performance  Computing 
and  Communications  (HPCC)  Program.  Development  of  the  NHSE  is  being  carried  out  by 
the  Center  for  Research  on  Parallel  Computation  (CRPC). 

MMF 

The  current  version  of  the  design  document  for  MMF  is  available.6  This  file  is  changed  as 
often  as  the  design  document  is  changed. 

The  heart  of  the  framework  is  a  C++  class  called  Evaluable,  which  encompasses  all  things 
that  can  be  evaluated  as  functions,  including  Models,  “Truth”  and  Memoizers  that  preserve 
function  values  to  avoid  expensive  recalculation.  Any  Evaluable  has  a  name,  knowledge  of 
its  number  of  inputs  and  outputs,  and  the  ability  to  have  a  subset  of  its  outputs  evaluated 
at  a  given  input.  It  is  intended  that  distinctions  between  objectives  and  various  kinds  of 
constraints  be  ignored  at  this  level — higher-level  “problem”  objects  know  which  components 
of  the  output  vector  are  objectives,  and  which  are  constraints. 

Memoizers  offer  the  additional  capability  of  reporting  whether  or  not  they  have  a  stored 
output  for  a  given  input,  and  will  offer  proximity  queries  including  functions  to  identify 
all  the  outputs  for  the  inputs  in  a  user-specified  box,  and  to  identify  closest-mat ches  to 
user-specified  input  points.  The  use  of  the  well-supported  Berkeley  DB  package,  along  with 
space-filling  curve  technology  to  minimize  disk  accesses,  will  make  this  an  efficient  solution 
to  the  archiving  problem. 

Models  offer  the  additional  capability  of  an  “update”  method,  in  which  an  input/output 
pair  from  “truth,”  or  from  a  more  accurate  model,  is  used  to  improve  the  current  model. 

Other  important  classes  in  the  implementation  are  ModeledOptProblems,  which  know 
which  parts  of  an  evaluable  output  are  objectives  and  which  are  constraints;  TrialSolutions , 
which  record  an  input,  the  corresponding  output,  and  a  radius  in  which  a  better  answer  is 

2  URL :  http : // extweb . nag . co . uk/ pro j ect s/PINEAPL . html 

3URL :  http : //www . nag . co . uk .numeric . html 

4URL:  http://Hww.nhse.org 

5NIST,  AHCPR,  DARPA,  NOAA,  NASA,  EPA,  VA,  NIH,  NSF,  ED,  NSA,  and  DOE. 

6URL:  http://www.caam.rice.edu/  dougm/MMF2. design.ps 
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likely  to  lie;  Terminators ,  which  decide  when  to  abandon  a  search;  Searches ,  which  recom¬ 
mend  points  that  might  improve  the  objective;  Polls ,  which  describe  optimization  fallback 
strategies;  and  various  utility  classes. 
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